Noisy media content encoding

ABSTRACT

Techniques are described for encoding noisy media content. A residual coefficient matrix representing differences in image content between a portion of a target image frame and portions of one or more reference frames can include noise within a high frequency band. Some of the noise can be removed by removing isolated residual coefficients under some estimated threshold. Additionally, the remaining noise can be reduced by attenuating their values selectively.

CROSS-REFERENCE TO RELATED APPLICATIONS

This disclosure is related to U.S. patent application Ser. No.15/041,606, titled NOISY MEDIA CONTENT ENCODING and filed on the sameday as the present patent application and incorporated by referenceherein and for all purposes.

BACKGROUND

Consumers have an ever-increasing array of options for consuming mediacontent, in terms of the types of media content (e.g., video, audio,text, etc.), providers of the media content, and devices for consumingmedia content. Media content providers are becoming increasinglysophisticated and effective at providing media content quickly andreliably to consumers.

Media content is typically encoded, which includes using compressionalgorithms to reduce the number of bits needed to transmit the mediacontent to a viewer's device where it is decoded for playback.Unfortunately, noisy media content can be difficult to encode,especially when noise can't be removed in a pre-processing stage.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of encoding media content with removal ofisolated non-zero high frequency residual coefficients and attenuationof high frequency residual coefficients.

FIG. 2 illustrates an example of a computing environment for encodingmedia content.

FIG. 3 is a flowchart illustrating an example of encoding media content.

FIG. 4 illustrates an example of determining a residual coefficientmatrix of a portion of an image frame.

FIG. 5 illustrates an example of removing isolated non-zero highfrequency residual coefficients and attenuating high frequency residualcoefficients.

FIG. 6 illustrates an example of adjusting quantization parameters.

FIG. 7 is a flowchart illustrating an example of adjusting quantizationparameters.

DETAILED DESCRIPTION

This disclosure describes media content encoding techniques for reducingnoise of media content. For example, in some standards such as HighEfficiency Video Coding (HEVC)/H.265, media content (e.g., movies,television shows, videos, etc.) is encoded, resulting in a compressedbitstream representing a version of that media content. The encodingprocess includes determining residual coefficients representing keptdifferences in image content between a portion (e.g., an M×N block ofpixels for a coding unit) of a target image frame and correspondingportions of reference frames. If the portion of the target frame orreference frame includes noise, then some of the residual coefficientsalso incorporate that noise. This results in some bits being used torepresent noise rather than meaningful image content. Moreover, if thenoise was not present, fewer bits could be used to encode the mediacontent, resulting in a more efficient compression technique.

In some implementations, an encoder reduces the number of bits beingused to represent noise by applying a Discrete Cosine Transform (DCT) totransform the residual coefficients from the spatial domain to thefrequency domain. Noise is often concentrated within the high frequencybands. The encoder analyzes the high frequency residual components andcompares their values to an estimated noise level of the portion of thetarget frame. If a residual coefficient is a non-zero value that isisolated from other residual coefficients by zero value residualcoefficients and its value is beneath a noise level threshold, then thenon-zero value residual coefficient is modified to be a zero valueresidual coefficient. Remaining high frequency residual components canbe attenuated by applying a custom quantization matrix that can be usedto reduce their values. As a result, high frequency residualcoefficients due to noise can be eliminated or reduced, and therefore,compressed into fewer bits during encoding.

This disclosure also describes media content encoding techniques forimproving the visual quality of media content that includes noise. Forexample, in standards such as HEVC, an image frame is segmented intocoding tree units (CTUs) which can be further partitioned into codingunits (CUs). The CUs can further be partitioned into prediction units(PUs). The residual coefficients for a portion of an image frame aredetermined for each CU. Different CUs in the image frame can bedifferent sizes, for example, some might be 4×4 pixels, some might be8×8 pixels, and so forth up to 64×64 pixels. During the encodingprocess, quantization parameters are applied to the residualcoefficients of each portion of the frame to scale down the residualcoefficients as a compression technique. However, if a much higher orlower quantization parameter is applied to the set of residualcoefficients of one CU than another set of residual coefficients ofanother CU, then the quality of the different portions of the imageframe can significantly vary, resulting in a noticeable flickeringacross the image frame when the media content is decoded and playedback.

In some implementations, the encoder can reduce the differences inquality by constraining the quantization parameters allowed to beapplied to the residual coefficients of the different portions. Forexample, the encoder determines the average quantization parameter forthe portions (or CUs) in the image frame. A quantization parameterwithin a scale around the average (e.g., −3 to +3 of the averagequantization parameter) can then be applied to the residual coefficientsof each of the portions of the frame. As a result, the quantizationparameter applied to different portions of the frame can be constrainedaround the average quantization parameter. This reduces the variancebetween the quantization parameters applied to the residualcoefficients, which reduces the flickering across the image frame.

As an example, FIG. 1 illustrates encoding media content with removal ofisolated non-zero high frequency residual coefficients and attenuationof high frequency residual coefficients. As shown in FIG. 1, image frame105 is a frame of a video with portion 115 including noise 120. Residualcoefficients 125 represent the differences in image content of portion115 to corresponding portions in one or more reference frames. That is,residual coefficients 125 provide data that can be used by a decoder toreconstruct image frame 105 from reference frames. The illustration inFIG. 1 is a simplified example. In some implementations, portion 115might be significantly smaller than shown, for example, on the order of4×4, 8×8, 16×16, 32×32, or 64×64 pixels.

In FIG. 1, residual coefficients 125 are represented by a matrix withthe coefficients in the frequency domain as the result of a DiscreteCosine Transform (DCT) applied to the residual coefficients in thespatial domain. The bottom-right portion of residual coefficients 125includes high frequency residual coefficients 130. In video and imagecontent, noise is often concentrated within high frequency residualcoefficients 130. More meaningful content (e.g., the image datarepresenting the basketball of portion 115 rather than noise 120) can bewithin the top-left portion of residual coefficients 125 (i.e., the lowfrequency), the top-right and bottom-left (i.e., the middle frequency),as well as possibly in the bottom-right high frequency residualcoefficients 130. If residual coefficients in high frequency residualcoefficients 130 are noise, then removing or attenuating them can resultin fewer bits being needed to encode portion 115, or more bits can bespent to encode the low and middle frequency residual coefficients toallow for an increase in quality.

A noise module can determine whether portion 115 includes noise usingnoise detection techniques. If portion 115 includes noise, meaning thatresidual coefficients 125 likely includes noise within high frequencyresidual coefficients 130, and the noise module determines an estimateof the noise level of that noise, then an encoder can analyze and modifyhigh frequency residual components 130 such that it needs less bits tobe encoded. The encoder can either zero out a residual coefficient orattenuate (e.g., reduce) a value to reduce the number of bits needed toencode high frequency residual components 130.

High frequency residual coefficients 130 can be zeroed out (i.e.,reduced from a non-zero value to a zero value) if they are isolated fromother non-zero residual coefficients and are within a threshold noiselevel range. For example, in FIG. 1, the residual coefficient with thevalue of 5 within high frequency residual coefficient is isolated fromCF_(14f), the residual coefficient with the value of 8 within highfrequency residual coefficients 130, and CF_(8f) by zero valuecoefficients. Moreover, if the noise level of portion 115 is 10, thenisolated high frequency residual coefficients that are within athreshold noise level range of half or less than the noise level (i.e.,5 or below) can be zeroed out. As a result, modified residualcoefficients 135 include modified high frequency residual coefficients145 with a 0 in the place of the 5 in high frequency residualcoefficients 130. This results in fewer bits needed to encode highfrequency residual coefficients 130.

High frequency residual coefficients 130 can also be attenuated (i.e.,reduced or divided by a factor). For example, the remaining highfrequency residual coefficients 130 can be attenuated by a factor byapplying (e.g., multiplying) a custom quantization matrix, as discussedlater, to residual coefficients 125 such that the remaining highfrequency residual coefficients 130 are attenuated. In the example ofFIG. 1, this results in the 8 value residual coefficient of highfrequency residual coefficients 130 being attenuated to a 4 in modifiedhigh frequency residual coefficients 145 (i.e., reduced by half). Thisalso results in fewer bits needed to encode high frequency residualcoefficients 130.

As a result, portion 155 of encoded image frame 150 includes similarvisual content as image frame 105 and portion 115, but without noise 120due to the modification of the residual coefficients of high frequencyresidual coefficients 130 to high frequency residual coefficients 145.

FIG. 2 illustrates an example of a computing environment for encodingmedia content. The computing environment of FIG. 2 includes media server210 which can be used to encode and provide media content to viewerdevices 205 a-e. Viewer devices 205 a-e can decode the encoded mediacontent for playback on a display screen. In FIG. 2, viewer device 205 ais a set-top box, viewer device 205 b is a laptop computer, viewerdevice 205 c is a smartphone, viewer device 205 d is a television, andviewer device 205 e is a desktop computer. However, relatedfunctionality may also be implemented within tablets, wearable devices,virtual or augmented reality headsets, video game consoles, etc.

Media server 210 can conform to any of a wide variety of architecturesand its functionality may be implemented by a variety of servers. Forexample, one server might encode media content and the encoded mediacontent might be provided to another server, for example, belonging to acontent delivery network (CDN). As such, the functionality andcomponents of media server 210 can use one or more servers and bedeployed at one or more geographic locations (e.g., across differentcountries, states, cities, etc.) using a network such as any subset orcombination of a wide variety of network environments including, forexample, TCP/IP-based networks, telecommunications networks, wirelessnetworks, cable networks, public networks, private networks, wide areanetworks, local area networks, the Internet, the World Wide Web,intranets, extranets, etc. Multiple entities may be involved in theencoding and delivery of media content and data related to the mediacontent, including content providers, internet service providers (ISPs),providers of content delivery networks (CDNs), etc. The functionalitydescribed herein also may be implemented by one or more differententities. For example, the functionality to provide playback of mediacontent can be integrated into a video player or software client undercontrol of one entity (e.g., on viewer devices 205 a-e), integrated intoa separate app from another entity, implemented in an edge server orcontent server of a CDN, a server of an ISP, etc.

It should be noted that, despite references to particular computingparadigms and software tools herein, the computer program instructionson which various implementations disclosed herein are based maycorrespond to any of a wide variety of programming languages, softwaretools and data formats, may be stored in any type of non-transitorycomputer-readable storage media or memory device(s), and may be executedaccording to a variety of computing models including, for example, aclient/server model, a peer-to-peer model, on a stand-alone computingdevice, or according to a distributed computing model in which variousfunctionalities may be effected or employed at different locations. Inaddition, reference to particular types of media content herein ismerely by way of example. Suitable alternatives known to those of skillin the art may be employed.

In FIG. 2, media server 210 can include various types of logic used toimplement encoder 215 to encode media content stored in media content220 and store encoded media content in memory such as encoded mediacontent 225, as well as playback handling logic 230 to provide requestedmanifest files from manifest data 235 to viewer devices 205 a-e so thatthey can request the encoded media content for playback. Media server210 can include one or more processors 240, memory, and other hardwarecomponents to encode media content and provide the encoded media contentto viewer devices 205 a-e. For example, processors 240 can executestored instructions in memory of media server 210 to implement encodingtechniques disclosed herein.

Viewer devices 205 a-e can also include various types of logic used toimplement a decoder to decode the encoded media content received frommedia server 210 and store the decoded media, as well as playbackhandling logic to request manifest files and store the manifest files,and request fragments of the encoded media content based on the manifestfiles. Viewer devices 205 a-e can include one or more processors,memory, and other hardware components to request and decode the encodedmedia content provided by media server 210. For example, the processorsof viewer devices 205 a-e can execute stored instructions in memory ofthe corresponding viewer device to implement decoding techniques basedon the encoding techniques disclosed herein.

A specific implementation will now be described with reference to FIG.1, the computing environment of FIG. 2, the flow diagram of FIG. 3, andthe examples of FIGS. 4 and 5. In FIG. 3, encoder 215 of media server210 selects a portion of an image frame and determines its residualcoefficients in a matrix corresponding to the size of the selectedportion (305).

FIG. 4 illustrates an example of determining a residual coefficientmatrix of a portion of an image frame. In some standards such asHEVC/H.265, an image frame (e.g., target image frame 405 in FIG. 4) issegmented into coding tree units (CTUs) which can be further partitionedinto coding units (CUs) for different portions of the image frame. Theresidual coefficients for a portion of an image frame are determined foreach CU. For example, portion 410 of target image frame 405 correspondsto a CU in a simplified example. Different CUs in the image frame can bedifferent sizes, for example, some might be a 4×4 block of pixels, somemight be 8×8 pixels, and so forth up to 64×64 pixels. By comparingportion 410 of target image frame 405 to one or more correspondingportions of reference frames 420 (e.g., frames in the media contentoccurring before or after target image frame 405 during playback),differences in the image content (or data) between portion 410 and theportions in reference frames 420 can be determined and expressed asresidual coefficients of a matrix with a similar size as portion 410.For example, if portion 410 is 4×4 pixels, then a 4×4 residualcoefficient matrix 415 represents the differences between the portionsas residual coefficients. Residual coefficient matrix 415 is used by adecoder of viewer devices 205 a-e to reconstruct target image frame 405to play back the media content.

In FIG. 4, portion 410 includes noise 425, which represents unwanted orundesirable image content. This affects the calculation of the residualcoefficients, and as a result, residual coefficient matrix 415incorporates data corresponding to the noise into the values of theresidual coefficients. If noise 425 was not present, then the values ofsome of the residual coefficients would be different. For example, someof the high frequency residual components would be 0, as discussedbelow.

Residual coefficient matrix 415 in FIG. 4 expresses the residualcoefficients in the spatial domain. The position within the matrixcorresponds to the pixel position within portion 410. In FIG. 3, theencoder can transform the residual coefficient matrix of the portion ofthe target image frame into the frequency domain (310). This can beachieved by the encoder applying (e.g., multiplying) a Discrete CosineTransform (DCT) to transform residual coefficient matrix 415 (in thespatial domain) of FIG. 4 into the frequency domain. For example, inFIG. 5, the encoder applies DCT 510 to residual coefficient matrix 415to generate frequency domain residual coefficient matrix 505.

Noise in image and video content is often concentrated within highfrequency bands. Therefore, much of the image data corresponding tonoise 425 of portion 410 in FIG. 4 is within the high frequency portion(i.e., the bottom-right quadrant corresponding to a 2×2 matrix) of the4×4 frequency domain residual coefficient matrix 505. More meaningful ornatural image content, such as the basketball in portion 410, isrepresented by the residual coefficients in the top-left portion orquadrant (representing low frequencies) of frequency domain residualcoefficient matrix 505, the top-right and bottom-left portions(representing the middle frequencies), as well as potentially the highfrequency portion.

Returning to FIG. 3, noise can be detected in the portion (325). Forexample, in FIG. 5, noise detection 515 determines whether portion 410in FIG. 4 includes noise by analyzing residual coefficient matrix 415 orfrequency domain residual coefficient matrix 505. Noise detection 515can be implemented by the encoder, or its analysis can be provided tothe encoder by a separate noise detector. If noise detection 515determines that portion 410 is noisy content, then it can determine orestimate a noise level for the portion. In FIG. 5, noise detection 515indicates that the noise level of portion 410 is a value of 10.

In some standards, such as HEVC, a quantization matrix is applied toresidual coefficient matrices to scale the residual coefficients as aform of compression. In HEVC, different quantization matrices arespecified for different sizes of the residual coefficient matrices. Ifno or little noise is detected in portion 410, then a standard, ordefault, quantization matrix as specified by the HEVC standard can beapplied by the encoder to frequency domain residual coefficient matrix505 (320). For example, if the detected noise level is beneath athreshold detected noise level, then the default quantization matrix isapplied.

By contrast, if noise is detected or is determined to be above athreshold detected noise level, then isolated high frequency residualcoefficients can be removed based on a noise level threshold (330). Thisresults in removing some noise from the portion.

In FIG. 5, frequency domain residual coefficient matrix 505 includeshigh frequency residual coefficients of 8 and 5. The residualcoefficient with the value of 5 is an isolated high frequency residualcoefficient because it is separated from other non-zero residualcoefficients by zero value residual coefficients. For example, ifCF_(14f) and CF_(8f) within the middle frequency bands are non-zerovalues, then they along with the residual coefficient with the value of8 in the high frequency band are separated from the residual coefficientwith the value of 5 by the zero value residual coefficients within thehigh frequency band. Often, isolated high frequency residualcoefficients are attributable to noise (e.g., noise 425 in FIG. 4)rather than meaningful image content (e.g., the basketball in portion410 in FIG. 4). In the example of FIG. 5, the residual coefficients tothe top and left of the residual coefficient with the value of 5 aredetermined to be 0 to identify an isolated high frequency residualcoefficient. However, in other implementations, the diagonal positioncan also be used to determine whether a residual coefficient isisolated, multiple Os might be identified (e.g., 2 columns and 2 rows ofOs isolating the high frequency residual coefficient), etc.

The encoder evaluates the identified isolated high frequency residualcoefficient and compares it with a noise level threshold based on thenoise level determined by noise detection 515. For example, if the noiselevel is 10, then the encoder can set a noise level threshold as beinghalf of the noise level and then remove, or zero out (e.g., make thevalue 0), any isolated high frequency residual coefficients if they arewithin the noise level threshold range at or beneath half of the noiselevel threshold. For example, 5 and below (to 0 or a negative number)can be the noise level threshold range. Since 5 is half of the noiselevel of 10 and within the noise level threshold range, the encoder caneffectively remove that noise image content by modifying the value of 5in frequency domain residual coefficient matrix 505 to be a zero value,as in first modified frequency domain residual coefficient matrix 525.This removes some of the noise content from the high frequency portionresidual coefficient matrix.

The encoder can also apply a custom quantization matrix to the residualcoefficients to attenuate high frequency residual coefficients (335).This can result in reducing some noise in the portion. For example, inFIG. 5, custom quantization matrix attenuation 530 is performed by theencoder to attenuate high frequency residual coefficients that were notzeroed out with isolated non-zero residual coefficient removal 520. Thiscan further reduce the contribution of noise accounted for by theresidual coefficients. The custom quantization matrix can scale the highfrequency residual coefficients differently than the standardquantization matrix used if no or little noise is detected. For example,the custom quantization matrix can attenuate, or reduce, the values ofthe high frequency residual coefficients more so than the standardquantization matrix such that the noise content within the residualcoefficients is reduced while the residual coefficients in the low andmiddle frequency bands are attenuated, or quantized, similar to thestandard quantization matrix.

The custom quantization matrix can scale down the remaining highfrequency residual coefficients by dividing them by a number,subtracting them with a number, rescaling them within a new range, etc.such that the residual coefficients are reduced to a non-zero value.Modifying, or reducing, the residual coefficients to a non-zero valueresults in reducing noise, but not removing it. This can be beneficialsince the values of the non-isolated residual coefficients are outsidethe noise level threshold range, and therefore, more care should beexercised since they are closer to the noise level. In FIG. 5, thisresults in second modified frequency domain residual coefficient matrix535 in which the high frequency residual coefficient with the value of 8being scaled down to a value of 3. When second modified frequency domainresidual coefficient matrix 535 is decoded, it provides portion 540 inFIG. 5, which is similar to portion 410 in FIG. 4 but includes no orreduced noise 425.

The technique can be repeated for the CUs of target image frame 405 andencoder 215 can store an encoded bitstream in encoded media content 225for image frames of the media content. Each application of quantizationmatrices for reducing and/or removal of high frequency residualcoefficients (as well as any additional encoding techniques), can bestored to provide an encoded image frame for the media content that canbe decoded for playback by a viewer device. Since many of the highfrequency residual coefficients would either be reduced or removed, theencoded version of the media content would have less noise.

Thus, by removing or attenuating the values of the residual coefficientsin the high frequency range, the contribution of noise can be reduced.This results in fewer bits being used for encoding noise, which canallow for fewer bits needed to encode portion 410 of target image frame405 of the media content. Therefore, more bits can be spent on moremeaningful image content to improve the visual quality of the playbackand/or fewer bits overall can be used to encode the media content.

The custom quantization matrix to be applied can be generated based onthe media content. For example, in two pass (or multi-pass) encoding,encoder 215 can analyze each CU and determine custom quantizationmatrices that can be applied to achieve the desirable attenuation innoise in a first pass. Several custom quantization matrices can begenerated in the first pass and can be selected in a second pass to beapplied to the residual coefficient matrices based on the distributionand values of the high frequency residual coefficients. In someimplementations, the custom quantization matrices can be the same sizeas the residual coefficient matrices. Multiple custom quantizationmatrices of the same size can be generated with different values for itscoefficients such that they attenuate the high frequency residualcoefficients differently based on how much noise is desirable to reduce.

In the HEVC/H.265 standard, a residual coefficient matrix is transformedinto the frequency domain and multiplied with a quantization matrix, aspreviously described. Additionally, a quantization parameter is alsoapplied to (e.g., multiplied or divided with) the residual coefficientmatrix to provide an additional and uniform scaling to each of theresidual coefficients.

As previously discussed, in standards such as HEVC/H.265, an image frameis segmented into coding tree units (CTUs) which can be furtherpartitioned into coding units (CUs). The residual coefficients for aportion of an image frame are determined for each CU. Different CUs inthe image frame can be different sizes. During the encoding process,quantization parameters are applied to the residual coefficients of eachportion, along with the quantization matrices as previously discussed.However, if a much higher or lower quantization parameter is applied tothe set of residual coefficients of one CU than to the set of residualcoefficients of another CU, then the quality of the different portionsof the image frame can significantly vary. This results in a noticeableflickering across the image frame when the media content is decoded andplayed back. This can be especially noticeable when the media contentincludes scenes with some portions having noisy content and someportions not having noisy content. As an example, a scene withcharacters watching old, grainy video on a television can result in someportions of the scene portraying the old, grainy video being noisy, butthe rest of the portions of the scene being relatively clear of noise.

FIG. 6 illustrates an example of adjusting quantization parameters. FIG.7 is a flowchart illustrating an example of adjusting quantizationparameters. FIGS. 6 and 7 provide an implementation in which thequantization parameters to be applied can be adjusted to reduce thevariance among them, and therefore, reduce the flickering due todifferences in quality of the portions.

In FIG. 7, portions of an image frame can be determined (705). Forexample, in FIG. 6, encoder 215 partitions target image frame 605 intoportions 610, 615, 620, and so forth. Each of the different portionscorrespond to a different CU, and therefore, different residualcoefficient matrices.

Under the HEVC standard, quantization parameters range from 0 to 51. Aquantization parameter difference of 6 between portions results in theportion associated with the higher quantization parameter doubling itsbits and therefore doubling its quality compared to the portionassociated with the lower quantization parameter. In FIG. 7, the averagequantization parameter for the portions can be determined (710) usingHEVC standard's scale of 0 to 51.

Higher quantization parameters can be applied (e.g., multiplied,divided, etc.) to difficult to encode portions than easier to encodeportions. For example, portions portraying a significant amount ofvarying image content, such as textures, hair, faces, etc. might beclassified as difficult to encode because those portions would need manybits to be able to encode the portion at an acceptable visual quality.By contrast, portions portraying an insignificant amount of varyingimage content, such as a flat wall or background, might be classified aseasy to encode because those portions would not need many bits to beable to encode the portions to an acceptable visual quality. As aresult, portions can be assigned a quantization parameter from 0 to 51based on the difficulty of encoding the portions. In FIG. 6, thisresults in portion 610 having a quantization parameter of 17, portion615 having a quantization parameter of 34, and portion 620 having aquantization parameter of 4. Portion 615 has a higher quantizationparameter because it includes a significant amount of meaningful imagecontent by portraying a large portion of the basketball from targetimage frame 605. In order to encode portion 615 to an acceptable visualquality level, a significant number of bits might need to be used toproperly display the textures and details of the basketball when portion615 is decoded for playback. Portion 610 has a lower quantizationparameter because it includes less detailed image content than frame605, and therefore, would need less bits to encode. Likewise, portion620 also has a lower quantization parameter because it corresponds toflat background detail. Other portions have their quantizationparameters in the range of 0 to 51 so that they are within that 52 valuerange.

Encoder 215 can determine a representative quantization parameter fortarget image frame 605, for example, the average quantization parameterof the portions across target image frame 605. In some implementations,the quantization parameter for a portion can be weighted based on itssize. For example, since portion 610 is four times larger (e.g., fourtimes as many pixels) than portion 615, the quantization parameter ofportion 610 might contribute “more” to the average quantizationparameter than the quantization parameter for portion 615. Thequantization parameters can be scaled by multiplying or dividing thembased on whether their corresponding portion is larger or smaller than areference portion. As an example, portion 615 might be an 8×8 portionand its quantization parameter of 34. Portion 610 is a 16×16 portionthat is four times larger than portion 615, and therefore, itsquantization parameter of 17 is scaled to a value of 68 (or maxed out at51 as that is the top of the range) when determining the averagequantization parameter for the portions of target image frame 605.

As a result, an average quantization parameter based on all of theportions of target image frame 605 is determined using the 0-51quantization parameter scale under HEVC. In the example of FIG. 6, anaverage quantization parameter for target image frame 605 is 12 and therange from the highest and lowest quantization parameters for theportions is 52 since the HEVC scale is from 0 to 51. In someimplementations, the average quantization parameter can be calculated ina first pass of the encoding process, as previously discussed. Next, inthe second pass of the encoding process, the quantization parameters tobe applied can be determined, as discussed below.

Encoder 215 can adjust the quantization parameters to be applied to theportions in target image frame 605 based on the average quantizationparameter (715). For example, in FIG. 6, the quantization parameters foreach of the portions can be within an adjusted quantization parameterrange of +/−3 of the average quantization parameter. That is, since theaverage quantization parameter is 12, the lowest quantization parameterthat can be assigned to a portion would be 9 and the highest would be15. This limits the variance between quantization parameters to 6, andtherefore, the highest quality portion can at most be twice as high of aquality than the lowest quality portion. The quantization parameterassigned to the portions under the 0-51 scale can be mapped or scaledproportionally to fit within the 9-15 scale. For example, if a portionis originally determined to have a quantization parameter of 26 (withinthe middle of the 0-51 range), that can be mapped to a quantizationparameter of 12 (within the middle of the adjusted quantizationparameter range of 9-15) by encoder 215. As another example, aquantization parameter of 50 might be mapped to an adjusted quantizationparameter of 15. The average quantization parameter can be at the centeror midpoint of the 9-15 scale (or range).

In the example of FIG. 6, a quantization parameter of 11 is to beapplied to portion 610, a quantization parameter of 13 is to be appliedto portion 615, and a quantization parameter of 9 is to be applied toportion 620. Since each of the portions corresponds to a quantizationparameter within the range of 9-12, the flickering due to the varianceof quantization parameters can be reduced, increasing the visual qualityof the playback.

In some implementations, the adjusted quantization parameter range canbe changed, for example, to increase or decrease the range from within+/−3 of the average quantization parameter. For example, if a largepercentage of portions have quantization parameters are determined bythe encoder to be concentrated at the lower end of the 0-51 quantizationparameter scale, then the adjusted quantization parameter scale mightrange from −4 from the average to +3 from the average. That is, anendpoint of the adjusted quantization parameter scale can be adjusted ifa certain percentage of quantization parameters using the larger 0-51HEVC scale are below a threshold quantization parameter.

By using an adjusted quantization parameter range of +/−3 of the averagequantization parameter, this limits the variance between the adjustedquantization parameters to a value of 6 between the lowest and highestvalues. This constrains the difference in quality between the lowestquality portion and the highest quality portion by a factor of 2. Thatis, the highest quality portion would at most be two times higher inquality (using a corresponding number of bits to encode) than the lowestquality portion, and therefore, reduces noticeable flickering acrosstarget image frame 605.

As previously discussed, the quantization parameter can be applied tothe residual coefficient matrix. For example, the residual coefficientmatrix corresponding to portion 610 can be multiplied by a quantizationmatrix and the adjusted quantization parameter during the encodingprocess.

While the subject matter of this application has been particularly shownand described with reference to specific implementations thereof, itwill be understood by those skilled in the art that changes in the formand details of the disclosed implementations may be made withoutdeparting from the spirit or scope of the invention. Examples of some ofthese implementations are illustrated in the accompanying drawings, andspecific details are set forth in order to provide a thoroughunderstanding thereof. It should be noted that implementations may bepracticed without some or all of these specific details. In addition,well known features may not have been described in detail to promoteclarity. Finally, although various advantages have been discussed hereinwith reference to various implementations, it will be understood thatthe scope of the invention should not be limited by reference to suchadvantages. Rather, the scope of the invention should be determined withreference to the appended claims.

What is claimed is:
 1. A computer implemented method comprising:determining a first residual coefficient matrix representing differencesbetween a first portion corresponding to a first coding unit (CU) of animage frame of media content and portions of one or more correspondingreference frames of the media content; transforming the first residualcoefficient matrix from a spatial domain to a frequency domain with aDiscrete Cosine Transform (DCT) to generate a frequency domain firstresidual coefficient matrix; determining noise information indicating afirst noise level corresponding to the first portion associated with thefrequency domain first residual coefficient matrix; removing noiseassociated with isolated residual coefficients in a high frequencyportion of the frequency domain first residual coefficient matrix basedon the first noise level to reduce noise in the first portion, eachisolated residual coefficient being adjacent to only zero value residualcoefficients and separated from non-zero residual coefficients by thezero value residual coefficients in the frequency domain first residualcoefficient matrix; reducing noise by applying a first quantizationmatrix to the frequency domain first residual coefficient matrixresulting in a quantized first residual coefficient matrix to attenuatenon-isolated residual coefficients in the high frequency portion of thefrequency domain first residual coefficient matrix, the non-isolatedresidual coefficients being adjacent to one or more non-zero valueresidual coefficients in the frequency domain first residual coefficientmatrix; and generating an encoded version of the media content using thequantized first residual coefficient matrix.
 2. The computer implementedmethod of claim 1, further comprising: determining a second residualcoefficient matrix representing differences between a second portioncorresponding to a second CU of the image frame of media content andportions of one or more corresponding reference frames of the mediacontent; transforming the second residual coefficient matrix from aspatial domain to a frequency domain to generate a frequency domainsecond residual coefficient matrix; determining noise informationindicating a second noise level corresponding to the second portionassociated with the frequency domain second residual coefficient matrix;determining that the second noise level corresponds to no noise beingwithin the second portion or little noise being within the secondportion; and applying a second quantization matrix to the frequencydomain second residual coefficient matrix to attenuate the residualcoefficients in a high frequency portion of the frequency domain secondresidual coefficient matrix, the second quantization matrix attenuatingresidual coefficients less than the first quantization matrix, and thesecond quantization matrix applied based on the second noise level. 3.The computer implemented method of claim 2, wherein removing noiseassociated with the isolated residual coefficients includes determiningthat the first noise level is within of a threshold detected noise levelrange, and the second noise level is outside the threshold detectednoise level range.
 4. The computer implemented method of claim 1,wherein attenuating the residual coefficients includes reducing valuesof the residual coefficients to non-zero values, and removing theisolated residual coefficients includes reducing values of the isolatedresidual coefficients to zero values.
 5. A system comprising: one ormore processors and memory configured to: remove noise associated withisolated residual coefficients in a high frequency range of a firstresidual coefficient matrix corresponding to a first portion of an imageframe of media content, each isolated residual coefficient beingadjacent to only zero value residual coefficients and separated fromother non-zero residual coefficients by the zero value residualcoefficients in the first residual coefficient matrix; and reduce noiseassociated with non-isolated residual coefficients in the high frequencyrange of the first residual coefficient matrix corresponding to thefirst portion of the image frame, the non-isolated residual coefficientsbeing adjacent to one or more non-zero value residual coefficients inthe first residual coefficient matrix.
 6. The system of claim 5, whereinremoving noise associated with the isolated residual data includesmodifying values of the isolated residual coefficients to be zerovalues, and reducing noise associated with the non-isolated residualcoefficients includes applying a first quantization matrix to the firstresidual coefficient matrix to reduce values of the non-isolatedresidual coefficients to non-zero values.
 7. The system of claim 6,wherein the one or more processors and memory are further configured to:apply a second quantization matrix to a second residual coefficientmatrix to reduce noise in a high frequency portion of a second residualcoefficient matrix associated with a second portion of the image frame,the second quantization matrix reducing residual coefficients less thanthe first quantization matrix.
 8. The system of claim 7, wherein the oneor more processors and memory are further configured to: determine thatnoise of the first portion is at a first noise level within a thresholddetected noise level range; and determine that noise of the secondportion is at a second noise level outside the threshold detected noiselevel range, being outside the threshold detected noise level rangeindicating that the second portion includes less noise than the firstportion.
 9. The system of claim 5, wherein the one or more processorsand memory are further configured to: determine a noise level of thefirst portion associated with the first residual coefficient matrix;determine a threshold noise level range based on the noise level,wherein values of the isolated residual coefficients are within thethreshold noise level range, and values of the non-isolated residualcoefficients are outside the threshold noise level range.
 10. The systemof claim 5, wherein the first portion corresponds to a first coding unit(CU) of the image frame.
 11. The system of claim 5, wherein the one ormore processors and memory are further configured to: generate anencoded version of the media content using the first residualcoefficient matrix based on the removed and reduced noise.
 12. Acomputer program product comprising one or more non-transitorycomputer-readable media having computer instructions stored therein, thecomputer program instructions being configured such that, when executedby one or more computing devices, the computer program instructionscause the one or more computing devices to: remove noise associated withisolated residual coefficients in a high frequency range of a firstresidual coefficient matrix corresponding to a first portion of an imageframe of media content each isolated residual coefficient being adjacentto only zero value residual coefficients and separated from othernon-zero residual coefficients by the zero value residual coefficientsin the first residual coefficient matrix; and reduce noise associatedwith non-isolated residual coefficients in the high frequency range ofthe first residual coefficient matrix corresponding to the first portionof the image frame, the non-isolated residual coefficients beingadjacent to one or more non-zero value residual coefficients in thefirst residual coefficient matrix.
 13. The computer program product ofclaim 12, wherein removing noise associated with the isolated residualdata includes modifying values of the isolated residual coefficients tobe zero values, and modifying the non-isolated residual coefficientsincludes applying a first quantization matrix to reduce values of thenon-isolated residual coefficients to non-zero values.
 14. The computerprogram product of claim 13, the computer program instructions furtherto cause the one or more computing devices to: apply a secondquantization matrix to a second residual coefficient matrix to reducenoise in a high frequency portion of a second residual coefficientmatrix associated with a second portion of the image frame, the secondquantization matrix reducing residual coefficients less than the firstquantization matrix.
 15. The computer program product of claim 14, thecomputer program instructions further to cause the one or more computingdevices to: determine that noise of the first portion is at a firstnoise level within a threshold detected noise level range; and determinethat noise of the second portion is at a second noise level outside thethreshold detected noise level range, being outside the thresholddetected noise level range indicating that the second portion includesless noise than the first portion.
 16. The computer program product ofclaim 12, the computer program instructions further to cause the one ormore computing devices to: determine a noise level of the first portionassociated with the first residual coefficient matrix; determine athreshold noise level range based on the noise level, wherein values ofthe isolated residual coefficients are within the threshold noise levelrange, and values of the non-isolated residual coefficients are outsidethe threshold noise level range.
 17. The computer program product ofclaim 12, wherein the first portion corresponds to a first coding unit(CU) of the image frame.
 18. The computer program product of claim 12,wherein the one or more processors and memory are further configured to:generate an encoded version of the media content using the firstresidual coefficient matrix based on the removed and reduced noise.